UNCLASSIFIED 


/ lepAoduced 
luf,  the 

•EFENSE  DOCUMENTATION  CENTER 


FOR 


SCIENTIFIC  AND  TECHNICAL  INFORMATION 


CAMERON  STATIC;;  VlEaA^RIA.  VIRGINIA 


UNCLASSIFIED 


NOTICE:  When  government  or  other  drawings,  speci¬ 
fications  or  other  date  are  used  for  any  purpose 
other  than  In  connection  with  a  definitely  related 
government  procurement  operation,  the  U.  S. 
Government  thereby  incurs  no  responsibility,  nor  any 
obligation  whatsoever)  and  the  fact  that  the  Govern¬ 
ment  may  have  formulated,  furnished,  or  in  any  way 
supplied  the  said  drawings,  specifications,  or  other 
data  is  not  to  be  regarded  by  implication  or  other¬ 
wise  as  in  any  manner  licensing  the  holder  or  any 
other  person  or  corporation,  or  conveying  any  rights 
or  permission  to  manufacture,  use  or  sell  any 
patented  Invention  that  any  In  any  way  be  related 
thereto. 


v-H 


GO 

t*0****’' 


L/JS-.  *5 

ft** 

/****%.  %*«**■ 
i':J 
CJ5 

7> 


-i 


RESEARCH  REPORT  30 
20  DECEMBER  1952 
*.E.R.  172-35 


401  402 


J 


SYMMETRIC  DUAL  NONLINEAR  PROGRAMS 


»rC  r  o 

.  g  5  <•*: 


by 

G.  B.  Dantzig,  E.  Eisenberg  and  R.  W.  Cottle 


OPERATIONS  RESEARCH  CENTER 


INSTITUTE  OF  ENGINEERING  RESEARCH 


UNIVERSITY  OF  CALIFOR  ill  A  -  BE  R'KELEY 


SYMMETRIC  DUAL  NONLINEAR  PROGRAMS 


by 

G.  B.  Dantzig,  E.  Eisenberg  and  R.  W.  Cottle 
Operations  Research  Center 
University  of  California,  Berkeley 


20  December  1962 


Research  Report  30 


This  research  has  been  partially  supported  by  the  Office  of  Naval  Research 
under  Contract  Nonr-222(83)  with  the  University  of  California.  Reproduce 
tion  in  whole  or  in  part  it  permitted  for  any  purpose  of  the  United  States 
Government. 


SYMMETRIC  NONLINEAR  DUAL  PROGRAMS 


I»  Introduction 

Most  known  results  on  duality  in  mathematical  programming  are  very 
closely  related  to  the  conditions  under  which 

(M)  max  min  K(x,  y)  =  min  max  K(x,  y) 

yeY  xeX  xeX  ycY 

where  the  sets  X  C  Rn  ,  Y  C.  Rm  and  the  function  K  :  X  x  Y  -*  R  .  As  an 
example,  the  duality  of  the  two  linear  programming  problems: 

i)  x  >  0  ,  Ax  >  b  ,  min  cx 
(1)  " 

ii)  y  >  0  ,  yA  <  c  ,  max  yb 

is  expressed  by  the  statement  (M)  with  X  =  R+  ,  Y  =  R^1  and 
K(x,y)  =  cx  +  yb  •  yAx  .  The  statement  (M),  or  the  equivalent  saddle -point 
statement,  avoids,  among  other  things,  one  pf  the  apparent  features  of  many 
duality  formulations,  i.  e. ,  asymmetry.  If,  for  instance,  the  primal  problem 
is  taken  to.  be 

(2)  x  e  Rn  ,  g^x)  <0  (i  =  1,  ...,m)  ,  min  gQ(x)  . 

where  gQ  ,  gj, . . . ,  gm  are  convex-differentiable  functions  of  x  e  Rn  ,  then 
a  formal  dual  is 


m 


(3) 


x  e  R 


ui± 


0  (i  =  1, . . .  ,m),  g^{x)  -  \  Ujg^x)  *  0 


max 


Ujg^x) 


where  gj(x)  is  the  gradient  of  g^  at  x  .  Apparently,  the  dual  problem  (3) 
is  not,  in  general,  of  the  same  form  as  the  primal  problem  (2).  Furthermore, 
while  the  objective  function  in  (2)  ia  assumed  convex,  the  objective  function 
in  (3)  may  be  neither  convex  nor  concave,  indicating  again  a  lack  of  symmetry. 
In  [3]  one  of  the  co-authors  of  this  note  discussed  the  situation  when  in  (M) 

X  «  R“  ,  Y  ■  8.“  ,  and  K  is  given  by: 

(4)  K(x,y)  =  f(x)  +  g(y)  -  yAx 

with  f,g  convex-homogeneous  and  f,  g,A  further  restricted  by  a  "feasibility" 
condition.  Results  of  Fenchel  [5]  (which  may  also  be  found  in  [7,  p.  227]) 
pertain  to  (M)  with  K  as  in  (4)  with  f,  g  convex  and  X,  Y  closed  convex 
sets.  The  main  result  of  this  paper  is  what  we  believe  to  be  a  new  duality 
formulation.  It  can  also  be  very  profitably  used  in  investigating  general  con¬ 
ditions  on  X  ,  Y  and  K  under  which  (M)  holds. 

Additional  or  related  efforts  in  the  area  of  duality  are  noted  in  the 
bibliography:,  they  include  Wolfe's  duality  theorem  [11,  Thm.  2]  which  states 
that  if  the  constraints  in  (2)  satisfy  the  Kuhn-Tucker  qualification  [8,  p.  483] 
and  if  Xq  solves  (2)  then  there  is  a  u  =  (uj, . .  .  »«m)  such  that  the  pair 
(xq,u)  solve  (3'  With  the  exception  of  special  cases,  such  as  quadratic 
programming,  the  converse  of  Wolfe's  theorem,  as  treated  independently  by 
Huard  [6]  and  Mangasarian  [9]  ,  requires  stronger  assumptions.  In  this 
paper  we  present  a  pair  of  symmetric  dual  programming  problems  related 
to  the  statement  (M)  .  In  a  way,  this  represents  an  extension  of  results 
obtained  by  one  of  this  paper's  co-authors  in  [1]  .  In  section  2  we  connect  the 
results  la  [1]  with  the  duality  theorem  which  is  established  ia  section  3. 


The  work  of  Dorn  [2}  on  quadratic  programming  exemplifies  the  fact 
that  the  formulation  of  section  1  for  primal  and  dual  problems  is  asymmetric. 
Dorn  considers  the  primal  problem  (the  sign  of  his  b  has  been  changed): 

(5)  Ax  +  b  >  0  ,  x  >  0  ,  min(^  x^Cx  +  p^x)  , 
and  its  dual  (with  a  slight  change  in  notation) 

(6)  -A^y  +  Cx  +  p  >  0  ,  y  >  0  ,  max(  -■£  x^Cx  -  b^y)  , 

where  C  is  a  symmetric  positive  semi-definite  matrix.  It  is  shown  in  [1] 
that  there  is  a  pair  of  naturally  symmetric  problems  related  by  duality.  They 
are: 


PRIMAL  PROBLEM 


Minimize 

IT  IT  T 

jY  Dy  +  jx  Cx  +  p  x 

(7) 

subject  to 

Dy  +  Ax  +  b  >  0 

and 

DUAL  PROBLEM 

x  e  Rj  ,  y  e  R“ 

Maximize 

1  T_  1  T  ,  T 

-fy  °y  ~  2 x  Cx-b  y 

(8) 

subject  to 

-A^y  +  Cx  +  p  >  0 

x  e  R“  ,  ye  R^1 

where  both  C  and  D  are  symmetric  positive  semi-definite  matrices.  In 
particular,  if  C  and  D  are  identically  sero,  then  (7)  and  (8)  reduce  clmply 
to  symmetric  dual  linear  programs. 
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II.  The  General  Symmetric  Duality  Statement 

Let  K(x,  y)  be  a  real- valued  and  twice  continuously  differentiable 
function  defined  on  an  open  subset  of  Rn+m  =  Rn  x  Rm  of  the  form  U  x  V 
where  U  is  open  in  Rn  and  V  is  open  in  Rm  .  We  define  Kj(x,  y)  to  be 
the  n-component  column  vector  which  is  the  gradient  of  K  with  respect  to 
ths  x-varlable  at  the  point  (x,  y)  ;  similarly,  K^x,  y)  is  the  m-component 
column  vector  representing  the  gradient  of  K  with  respect  to  y  at  the 
point  (x,  y)  e  U  x  V  .  We  introduce  a  double  subscript  notation  for  the 
matrices  of  second  partials.  Thus, 


K2«x.y,  - 

Kn<*'V>  *  ! 


The  dual  programming  problems  we  are  primarily  Concerned  with 
in  this  paper  may  be  stated  in  general  as: 


PRIMAL  (P) 

DUAL  (P*) 

Min  [  K(x,  y)  -  yT  K2(x,  y)  ] 

*»y 

Subject  to:  K2(x,  y)  <  0 

—  XX  _  XXX 

x  «  R+  ,  y  <  R+ 

Max  [  K(x,  y)  -  x*  Kj(x,  y)  ] 
x,y 

Subject  to:  K^x,  y)  >  0 

x  e  Rj  ,  y  €  R™ 

as  a  special  case,  with  K  defined  by 


(10)  K(x,  y)  =  ^  xTCx  +  pTx  -  yTCy  -  b^y  -  yTAx 

the  primal  and  dual  problems  above  reduce  to  the  quadratic  case  (7)  and  (8). 

The  general  strict  case  will  consist  of  the  two  problems  (P)  and 
(P*)  with  K  having  the  following  properties: 

(i)  K  is  real  valued  on  the  cartesian  product  U  x  V  (briefly, 

K  :  U  x  V  -*  R)  where  U  and  V  are  open  subsets  of  Rn 
and  Rm  respectively,  such  that  R^  C  U  ,  R™  C  V  . 

(ii)  K  is  twice  continuously  differ  entiable  on  U  x  V  • 

(iii)  for  each  fixed  x  e  r"  ,  K  is  strictly  concave  in  y  . 

(iv)  for  each  fixed  y  e  R™  ,  K  is  strictly  convex  in  x  . 

Observe  that  if  we  omit  "strictly"  in  conditions  (iii)  and  (iv)  then  with  K  as 

given  by  (10)  for  the  quadratic  case,  all  the  other  conditions  are  satisfied. 

The  general  strict  case  and  the  quadratic  caBe  are,  of  course,  not  mutually 

exclusive;  specifically,  the  quadratic  case  becomes  an  instance  of  the  other 

providing  C  and  D  are  both  positive  definite  (hence,  nonsingular). 

We  find  it  convenient  to  use  a  single  symbol  for  the  objective 
* 

functions  in  (P)  and  (P  ): 

$(x,y)  =  K(x,  y)  *  yTK,(x,  y) 

(ID  ,  T 

r?(x,  y)  =  K(x,  y)  -  x*Kj(x,y)  . 

Also,  we  denote  by  P  the  set  of  all  pairs  (x,  y)  satisfying  the  constraints  of 

i 

(P)  ,  i.  e. ,  x  €  R^  ,  y  e  R^1  ,  K^x,  y)  <  0  ;  similarly,  P  denotes  the  set 
of  all  pairs  (x,  y)  satisfying  x  e  •  y  c  R™  ,  and  K^(x,  y)  >  0  . 


In  the  next  section  we  state  precisely  what  the  duality  relation 

between  (P)  and  (P  )  is.  At  this  point,  it  should  be  remarked  that  neither 

* 

of  the  problems  (P)  and  (P  )  need  be  a  convex  programming  problem. 

For  example,  in  (P)  the  objective  or  the  constraint  functions  may  fail  to  be 
convex,  and  analogously  in  (P  )  .  Furthermore,  it  should  be  noted  that  the 
duality  requirements  defined  in  the  next  section  are  of  the  type  which  will 
not  hold  unless  some  regularity  conditions  are  imposed  on  the  function 
K(x,  y)  .  In  view  of  the  possible  absence  of  convexity  (or  concavity),  the 
Kuhn-Tucker  theorem  [8,  p.  84]  on  necessary  conditions  of  optimality 
seems  appropriate  to  obtain  the  desired  duality  relations.  Thus,  it  appears 
that  the  constraint  qualification  (see  Appendix)  is  a  suitable  regularity 
assumption.  Indeed,  this  assumption  is  sufficient. 
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We  shall  say  that  a  relation  of  duality  holds  between  (P)  and  (P*) 
providing  that  the  following  two  conditions  hold: 

(a)  sup  rj(x»y)  <  inf  5(x*y) 

(x,  y)eP*  (x,  y)eP 

and 

£ 

(b)  (P)  is  solvable  if,  And  only  if,  (P  )  is  solvable,  in  which  case 

max  rj  a  min  5 

e 

P  P 

(Solvable  means  "has  an  optimal  solution.") 

REMARK:  the  further  condition: 


(c)  If  exactly  one  of  the  programs  is  feasible  then  its  objective 
function  is  unbounded  in  the  direction  of  optimization; 

(Feasible  means  that  there  exist  variables  satisfying  its  constraints) 

which  holds  for  dual  quadratic  or  linear  programs  (i.  e. ,  in  the  quadratic  case, 

see  [  1,  theorem  3  ]  need  not  hold  in  the  strict  case,^  This  may  be  seen  from 

the  example:  K(x,  y)  =  ex  -  e*y  {  in  this  case  (P*)  is  trivially  feasible  while 

(P)  is  infeasible;  however  rj(x,  y)  »  ex  -  e  ^  -  xex  which  is  bounded  above 

for  (x,  y)  >  0  .  We  shall  show  that  in  the  strict  case^a  relation  of  duality 

$ 

hold 8  between  (P)  and  (P  )  providing  they  satisfy  a  constraint  qualification; 
we  begin  by  showing  (a): 

THEOREM  1:  sup  q  <  inf  { 

P*  P 

^i.  e. ,  in  the  general  strict  case. 


"■4*  _  '^r  "  -l _ _ _  '-■' 


Proofs  Uiiag  the  convention 

e 

■up  n  =  -00  if  P  it  empty 
P* 

inf  $  =  +oo  if  P  it  empty 

P 

we  need  only  show  that  if  (x,y)eP  and  (x,y)eP  then  n(x,y)  <  5(x,y). 

e 

By  definition  of  P  and  P  we  have 

K,(x,y)  <0,  x  >  0,  y  >  0 

(12) 

K^y)  >0,  x  >  0.  y  >  0  . 


Thut: 

(13)  yTK2(x,y)  <  0.  xTKj(x,y)  >  0 


and 

(14)  yTK2(x,y)  -  xTKI(x,y)  <  0  . 


Since  K  it  convex-concave  and  differentiable  we  have  (tee  [8]  Vol.  l,p.405, 
No.vii) 

xTK1(x,y)  -  xTK1(x,y)  <  K(x,y)  -  K(x,y) 

(15) 

yTK2(x,y)  -  yTK2(x,  y)  <  K(x,y)  -  K(x,y) 

Adding  the  two  inequalitiet  of  (15),  rearranging  terma,  and  uting  (14),  we 
obtain: 

»Kx.y)  ■  K(x,  y)  -  xTKj(x,  y)  < 

<  K(x,y)  -  yTK2(x,  y) +  |^yTK2(x,y) -xTKj(x,y)  J  < 

<  K(x,  y)  -  yTK2(x,  y)  =  f(x.y)  , 
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completing  the  proof  of  Theorem  1. 

NOTE:  The  proof  of  Theorem  1  doe*  not  require  the  "atrictnees"  a  a  sumption 
of  (iii),  (iv);  only  convexity-concavity  and  differentiability  are  essential. 

From  Theorem  1  follows  the  obvious  remark  that  if  (x0>y0)  e  P  ,. 
(x(j,y0)  €  P*  have  the  property  that  5(xQ,  yQ)  <  n(x0>  yQ)  then  (xQ,  yQ)  and 
<3L.?0>  are  optimal  solutions  of  P  and  ‘ P*  respectively.  This  will  be  used 
in  proving: 

THEOREM  2  (Duality  Theorem):  Assume  that  P  and  P*  satisfy  the  Kuhn- 
Tucker  constraint  qualification.  Then  (P)  is  solvable  if,  and  only  if,  (P  ) 
is  solvable,  in  which  case 

max  H  =  min  f 
P*  P 

Proof:  We  shall  show  that  if  (x,y)  solves  (P*)  then  (x,  y)  also  solves 
(P)  and  5(x,y)  *  t)(x,y)  (the  proof  of  the  converse  is  analogous).  Consider 
the  function: 

<Kx,  y.u)  =  K(x,  y)  -  xTKj(x,y)  +  ttTK1(x,y) 

By  the  Kuhn-Tucker  optimality  conditions  [8,  theorem  1]  ,  and  since  (x,y) 
solves  (P*)  ,  there  exists  a  vector  u  e  Rn  satisfying  the  following: 

(16)  ♦jfx.y.u)  =  K^fcyHu  -  x)  <  0 

(17)  *2(x, y.u)  =  K2(x,y)  +  K21(x,y)(u  -  x)  <  0 

(18)  y,u)  =  Kj(x,  y)  >  0 

(19)  x^  <Pj(x,y,u)  +  yT^2(x,y,u)  = 

»  xTKu(x,y)(u  -  x)  +  yTK2(x,y)  +  yTK21(x,y)^  -  x) 
■  0 
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i 

i 

) 

i 

(20)  tt^x.y)  =  0 

(21)  u  >  0,  x  >  0,  y  >  0  . 

It  then  follows  that: 

(22)  xTK11(x,  y)(u  -  x)  =  0 

(23)  yTK2(x,y)  +  yTK2  ^x.yMu-  x)  =  0 


From  (16)  and  (21)  we  get: 

(24)  uTK11(jE,y)(u  -  x)  <  0 
and  thus: 

(25)  (u  -  xjTK11(x,y)(u  -  x)  <  0  . 

But  because  K  is  strictly  convex  in  x,  it  follows  that  is  a  positive 
definite  matrix,  thus  u  -  x  =  0,  i.e. , 

(26)  u  =  x 

It  then  follows  from  (17)  and  (26)  that  the  pair  (x,  y)  is  feasible  for  (P). 
From  (26)  and  (19)  it  follows  that 

(27)  yTK2(5,y)  =  0  , 
while  from  (26)  and  (20)  ws  obtain: 


(28)  xTK1(x,y)  =  0  . 

We  conclude  that  (x,y)cP*P*  and  ?(x,  y)  =  »Kx,y)  . 


Qe  E*  D 
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We  state  two  immediate  corollaries: 


COROLLARY  1: 

If  either  program  (P)  or  (P  )  is  solvable  then  there  is  a  joint  eolation 
(i.e.,  a  pair  solving  both  programs). 

Proof:  It  is  clear  from  the  proof  of  Theorem  2  that  in  the  general  strict 
case  the  optimal  solutions  of  (P)  and  (P*)  are  the  same*  For  the  quadratic 
case  see  [  1,  Theorem  !]• 

COROLLARY  2: 

If  (x,  y)  is  a  joint  solution  of  (P)  and  (P*)  then  yT^tx,  y)  =  0  =  x^K^(x,  y)  . 

Proof:  For  the  general  strict  case  note  Equations  (27)  and  (28);  for  the 
quadratic  case  see  the  remark  after  Theorem  4  in  [1] . 


APPENDIX 


The  Kuhn-Tucker  Constraint  Qualification 

T 

Suppose  F(x)  =  [  f  j(x),  •  •  •  •  fm(x)  ]  where  each  is  an 

everywhere  differentiable  function  of  x  >  0  .  Let  x°  be  such  a  point.  Taking 
the  partial  derivatives  of  the  functions  f^  at  x°  ,  We  define 


which  is  regardeid  as  an  m  x  n  matrix,  the  i**1  row  of  which  is  the  gradient 
(transposed)  of  the  function  f^  evaluated  at  x°  . 

Let  x°  be  a  boundary  point  of  the  (constraint)  set  C  =  {xfR'  )f(x)  >0}. 
Separate  the  inequalities  F(x°)  >  0  ,  Ix°  >  0  (where  I  is  the  m  x  n  identity 
matrix)  into 

Fj(x°)  =  0  ,  Ijx°  *  0  ,  F2(x°)  >0  ,  I2x°  >  0  . 

That  is,  consists  of  the  subset  of  function^  in  F  which  vanish  at  x°  , 

and  F2  consists  of  those  which  are  positive  at  x°  .  Ij  and  I2  are  diagonal 
matrices  having  ones  and  zeros  as  diagonal  entries.  In  particular,  Ij  has 
ones  in  those  rows  where  the  components  of  x°  are  zero,  and  has  zeros  on 
the  rest  of  the  diagonal.  12  has  ones  in  the  rows  where  x°  has  positive 
components  and  has  zeros  on  the  remainder  of  its  diagonal. 

The  Maximum  Problem  of  Kuhn  and  Tucker  [  8,  p.  483  ]  is:  Maximize 
a  differentiable  function  g(x)  constrained  by  F(x)  >  0  ,  x  >  0  .  Further 
restrictions  on  the  constraint  set  are  required  in  order  to  make  certain  con¬ 
clusions  about  an  optimal  solution  of  the  maximum  problem.  These  restrictions 
ars  embodied  in  the  constraint  qualification  which  we  now  stats. 
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For  every  boundary  point  x°  of  C  ,  any  vector  differential  dx 
satisfying  the  homogeneous  linear  inequalities 

F°dx  >  0  ,  Ijdx  >  0 

is  tangent  to  a  differentiable  arc  contained  in  C  .  That  is,  to  every  dx 
satisfying  the  above  inequalities,  there  corresponds  an  arc  x  a  a(Q)  , 

0  <  0  <  1  ,  with  x°  »  a(0)  ,  such  that  [  da/dO  ]  a  Xdx  for  some  positive 
scalar  \  . 
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